The Trouble with Machine Translation Coherence

This paper introduces the problem of measuring coherence in Machine Translation. Previously, local coherence has been assessed in a monolingual context using essentially coherent texts. These are then artificially shuffled to create an incoherent one. We investigate existing models for the task of measuring the coherence of machine translation output. This is a much more challenging case where coherent source documents are machine translated into a target language and the task is to distinguish them from their human translated counterparts. We benchmark stateof-the-art coherence models, and propose a new model which explores syntax following a more principled method to learn the syntactic patterns. This extension outperforms existing ones in the monolingual shuffling task on news data, and performs well in our new, more challenging task. Additionally, we show that breaches in coherence in the translation task are much more difficult to capture by any model.